TINY-11177: Vastly improve remote testing #145

TheSpyder · 2024-08-16T06:37:00Z

Related Ticket: TINY-11177

Description of Changes:

Remote testing was slow because bedrock sent session POST requests to /start before each it block, and then again to /results after each it block. It would wait for these requests to finish before continuing.
/start is now sent at startup only.
/results is now only sent for failure and skip, or every 30 seconds if tests are passing happily.
The bedrock UI doesn't block on these status requests anymore, it just throws them into an array and uses Promise.all() at the very end to wait for them.
Unfortunately the bedrock CLI console HUD depended on receiving these status updates, so I had to adjust it to account for the missing reports.
The bedrock HTML page now has a fake icon to remove the favicon.ico request which probably also slowed things down.
A proper keep-alive header is now set, instead of using the default 5 seconds which was causing occasional 502 Bad Gateway errors.
Reverted TINY-10708: Suppress bedrock logs in remote testing #142 which was designed to solve this problem but server-side was the wrong answer.

Pre-checks:

Changelog entry added
package.json versions have not been changed (done by Lerna on release)
~~Tests have been added (if applicable)~~

Before merging:

Ensure internal dependencies are on appropriate versions
- For stable releases, all dependencies must be stable
- For release candidates, all dependencies must be release candidates or stable

spocke

How does this affect junit reporting would the in flight requests make them in random order or would things be skipped.

TheSpyder · 2024-08-19T00:05:47Z

I posted in slack that this destroys JUnit reporting. Ideas for how to fix:
https://cksource.slack.com/archives/C04KXE6QUTX/p1724022794632699?thread_ts=1723766938.410059&cid=C04KXE6QUTX

modules/runner/src/main/ts/reporter/Reporter.ts

Jenkinsfile

modules/runner/src/main/ts/reporter/Reporter.ts

…avicon.ico`

…it for the report AJAX requests to complete before moving on to the next test.

…it would be misleading if they were treated as passing.

…th test reports and offsets.

…ts. Manual mode works, auto still needs some work.

…to the end of the test run are now waited for correctly.

…ce for every result

…an empty rejected promise

… nodejs.

…emoving it is a pain. Use es2019 instead.

…ote driver once every 4 minutes

…stopOnFailure in auto mode.

…p with identification (and already found a bug in tinymce)

…sily identify them

… elements it added

ltrouton

Not an expert on bedrock so don't know everything I am looking at but this looks like it will be a big improvement 👍

modules/runner/src/main/ts/reporter/Reporter.ts

spocke

Looks good, lots of nice improvements beyond the perf things. 👍

modules/runner/src/main/ts/reporter/Callbacks.ts

modules/runner/src/main/ts/reporter/Reporter.ts

modules/runner/src/main/ts/runner/Runner.ts

modules/runner/src/main/ts/runner/TestRun.ts

modules/server/src/main/ts/bedrock/auto/RemoteDriver.ts

modules/server/src/main/ts/bedrock/server/Apis.ts

modules/server/src/main/ts/bedrock/server/Controller.ts

modules/server/src/main/ts/bedrock/server/EffectUtils.ts

modules/server/tasks/bedrock.js

TheSpyder · 2025-01-24T08:09:48Z

Resolved all except one which I'll deal with on Monday and then release this 👍

…export I'll never know

…o simplify the code that calls it. Removed unnecessary export declarations, which means we no longer need a safety-net `checkSiblings` call in `runSuites` which will by definition never pass the root test suite condition.

TheSpyder · 2025-01-27T01:14:22Z

I'm glad to finally finish this one ☺️

TheSpyder requested a review from tjdett August 16, 2024 06:37

TheSpyder requested a review from a team as a code owner August 16, 2024 06:37

TheSpyder requested review from spocke, hamza0867, ltrouton and jscasca August 16, 2024 06:37

spocke approved these changes Aug 16, 2024

View reviewed changes

TheSpyder marked this pull request as draft August 19, 2024 00:05

TheSpyder force-pushed the feature/TINY-11177 branch from 4feac22 to e80663d Compare September 15, 2024 23:54

ltrouton reviewed Sep 17, 2024

View reviewed changes

modules/runner/src/main/ts/reporter/Reporter.ts Show resolved Hide resolved

Jenkinsfile Outdated Show resolved Hide resolved

modules/runner/src/main/ts/reporter/Reporter.ts Outdated Show resolved Hide resolved

ltrouton self-requested a review September 17, 2024 23:23

TheSpyder and others added 18 commits December 4, 2024 11:13

TINY-11177: Add an empty favicon to prevent the browser requesting `f…

f9e89be

…avicon.ico`

TINY-11177: Add keep-alive header to stop weird 502 bad gateway errors

b73ba9a

TINY-11177: Send significantly less reports from the client. Don't wa…

0621111

…it for the report AJAX requests to complete before moving on to the next test.

TINY-11177: Rebase conflicts

7c24b29

TINY-11177: Re-implemented skip reports, we have enough of them that …

8e05635

…it would be misleading if they were treated as passing.

TINY-11177: Updated reporter test for logical changes. Fixed a bug wi…

72d4f7f

…th test reports and offsets.

TINY-11177: Removed @ephox/wrap-promise-polyfill

f277686

TINY-11177: Rewriting the entire report system to support batch resul…

5338432

…ts. Manual mode works, auto still needs some work.

TINY-11177: Delete route that accepts a single test result

54b707f

TINY-11177: Changelog

2ad5cae

TINY-11177: Fixed test

facf52c

TINY-11177: Improved test failure reporting

7a9892e

TINY-11177: The test wasn't broken, my code was. Messages sent close …

ef03603

…to the end of the test run are now waited for correctly.

TINY-11177: Only update the HUD once for batch results, instead of on…

b1c5c5f

…ce for every result

TINY-11177: Fixed issue where it was not clear test failures must be …

f8b7565

…an empty rejected promise

TINY-11177: Send start once, not once per page

ac820f8

TINY-11177: Fix HUD test count now that we are sending all results

fd9bb6a

TINY-11177: Only record results if there are results

872dca2

TheSpyder added 13 commits January 17, 2025 14:56

TINY-11177: That's a no to Node 22. Downgrade rollup to work on older…

fc2c431

… nodejs.

TINY-11177: es2022 is too high for esm, which is a dead project and r…

2914e70

…emoving it is a pain. Use es2019 instead.

v15.0.0-alpha.7

5cf4b07

TINY-11177: Extend idle LambdaTest timeout to 5 minutes

7911a13

v15.0.0-alpha.8

d018041

TINY-11177: Make sure there is at least some interaction with the rem…

81e54aa

…ote driver once every 4 minutes

v15.0.0-alpha.9

44518a7

TINY-11177: Verbose log bedrock settings. Allow settings to override …

bcc33ce

…stopOnFailure in auto mode.

TINY-11177: Enforce that tests are in describe blocks - this will hel…

0629f21

…p with identification (and already found a bug in tinymce)

TINY-11177: Add describe block wrappers to old style tests to more ea…

b84f05a

…sily identify them

TINY-11177: Add checks to fail a test file if it doesn't clean up DOM…

d322a0f

… elements it added

TINY-11177: Bump default chunk to 2000

b9bd5a0

v15.0.0-alpha.10

f793b47

TheSpyder marked this pull request as ready for review January 21, 2025 05:22

TheSpyder requested a review from spocke January 21, 2025 05:23

TheSpyder mentioned this pull request Jan 21, 2025

TINY-11177: Performance improvements to CI testing tinymce/tinymce#10104

Merged

6 tasks

ltrouton approved these changes Jan 23, 2025

View reviewed changes

modules/runner/src/main/ts/reporter/Reporter.ts Outdated Show resolved Hide resolved

modules/runner/src/main/ts/reporter/Reporter.ts Show resolved Hide resolved

TINY-11177: Fix import spacing

902eab3

TheSpyder mentioned this pull request Jan 23, 2025

TINY-11431: Add chrome clipboardPermission handle #153

Draft

4 tasks

TheSpyder requested review from tiny-james and Skylite73 January 24, 2025 06:04

spocke approved these changes Jan 24, 2025

View reviewed changes

TheSpyder added 2 commits January 24, 2025 17:34

TINY-11177: Upgrade lambdatest and AWS library dependencies

e4681f3

TINY-11177: Review fixes

d615cab

TheSpyder added 2 commits January 24, 2025 18:12

TINY-11177: Why we patch sourcemapped-stacktrace to remove the ES6 …

b5c117e

…export I'll never know

TheSpyder merged commit 18ed8c5 into master Jan 27, 2025
5 checks passed

TheSpyder deleted the feature/TINY-11177 branch January 27, 2025 01:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TINY-11177: Vastly improve remote testing #145

TINY-11177: Vastly improve remote testing #145

TheSpyder commented Aug 16, 2024 •

edited

Loading

spocke left a comment

TheSpyder commented Aug 19, 2024

ltrouton left a comment

spocke left a comment

TheSpyder commented Jan 24, 2025

TheSpyder commented Jan 27, 2025

TINY-11177: Vastly improve remote testing #145

TINY-11177: Vastly improve remote testing #145

Conversation

TheSpyder commented Aug 16, 2024 • edited Loading

spocke left a comment

Choose a reason for hiding this comment

TheSpyder commented Aug 19, 2024

ltrouton left a comment

Choose a reason for hiding this comment

spocke left a comment

Choose a reason for hiding this comment

TheSpyder commented Jan 24, 2025

TheSpyder commented Jan 27, 2025

TheSpyder commented Aug 16, 2024 •

edited

Loading